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ASSOCIATE  RATINGS  AND  SENIOR  SERVICE 
SCHOOL  SELECTION 

/ 

I  BACKGROUND 


^Interest  in  the  use  of  associate  ratings  as  a  technique  for 
officer  selection  has  undergone  a  recent  revival  in  the  A -my. 
Previous  studies  have  demonstrated  the  validity  of  ratings  made  in 
precommissioning  programs  for  later  active-duty  officer  performance. 
A  previous  effort  with  Army  colonels  had  demonstrated  the  feasibil¬ 
ity  and  reliability  of  associate  ratings  at  higher  grade  levels. 
This  study  was  an  attempt  to  further  demonstrate  its  usefulness  at 
higher  command  levels.  \ 


In  the  fall  of  1970  a  student  project  (supervised  by  LTC  D.  M. 
Malone)  at  the  Army  War  College  (AWC)  was  developed  to  test  the  util¬ 
ity  of  associate  ratings  in  selecting  officers  suitable  for  attend¬ 
ance  at  senior  service  colleges.-'  In  the  spring  of  1971,  AWC  pro¬ 
vided  the  data  to  the  Army  Research  Institute  (ARI)  for  additional 
analysis. 


\ 


OBJECTIVES  OF  THE  STUDY 


The  primary  objective  was  to  determine  the  reliability  of 
associate  ratings  and  the  relationship  between  ratings  and  board 
action  for  senior  service  college  selection.  An  additional  factor 
of  two  rater  samples  at  different  levels  of  military  schools,  AWC, 
peers,  and  Command  and  General  Staff  College  (C&GSC)  subordinates 
was  studied  to  determine  differential  results.  .■ 


L/  Medland,  F.  F.,  and  Smith, K.  Associate  ratings  of  senior  officer 
potential.  ARI  Research  Problem  Review,  in  press. 
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— '  Brock,  L.  A.,  and  Wardrop,  D.  H.  An  examination  of  the  selective 
use  of  peer  ratings  in  officer  selection  procedures.  U.  S.  Army 
War  College:  Carlisle  Barracks,  Pa.  March  1971. 


METHOD  OF  ATTACK 


RATING  SCALES 

Raters  were  asked  to  rate  the  ratee  on  two  5-point  scales,  degree 
of  acquaintanceship  and  suitability  for  attendance.  A  rater  was  in¬ 
structed  to  go  through  the  roster  and  mark  the  names  of  all  officers 
he  was  reasonably  sure  he  recognized,  then  rate  each  of  the  officers 
on  each  of  the  two  scales  (see  Appendix  A  for  printed  instruction  sheet) . 
Three  scores  were  then  derived  for  each  officer. 

.  n 

1)  Acquaintanceship:  A  =  Y',  a 
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n 

n 

2)  Suitability:  S  “  S  8 

1 

n 

£  (s-3)  (.5)  (a+l)+6 

3)  Composite:  C=  — - - - 1 - 

where  a  ■  rated  degree  o.f  acquaintanceship 
s  ■  rated  suitability 
n  =  number  of  raters 

The  rationale  for  the  composite  score  was  that  when  acquaintance¬ 
ship  is  high  (level  5)  the  range  of  composite  scores  is  greatest  (0  to 
12);  with  intermediate  acquaintanceship  (level  3),  the  range  is  inter¬ 
mediate  (2  to  10) ;  with  low  acquaintanceship  (level  1) ,  the  range  is 
least  (4  to  8). 


SAMPLE 

Ratees.  The  roster  used  was  made  up  of  all  infantry  officers 
eligible  to  attend  a  senior  service  college  for  the  academic  year  of 
1971-1972  (N  ■  1978).  For  the  analysis  only,  this  group  was  further 
broken  down  into  three  groups  based  on  selection  board  action:  a 
group  not  considered  by  the  selection  board  (N  *  1717);  those  considered 


i. 
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by  the  selection  board  but  not  selected  (N  =  164);  and  a  group  of 
officers  selected  by  the  board  (N  =  95).  The  selection  board  rank- 
ordered  each  of  the  selected  officers  from  1  to  95,  with  1  the  most 
preferred  officer. 

I 

Raters.  Two  groups  rated  the  officers  on  the  roster.  One  (N  =  57) 
was  composed  of  all  infantry  officers  attending  the  Army  War  College 
during  1970-71.  Sample  two  (N  =  235)  consisted  of  all  infantry  officers 
attending  Command  and  General  Staff  College  during  the  same  period. 


METHOD  OF  ANALYSIS 

The  reliabilities  of  the  ratings  were  estimated  using  the  generalized 
formula  for  the  reliability  of  averages: 


where 

2 

<r^  =  individual  variance 

=  rating  variance 
rii  =  reliability 

3. 

This  index  is  generally  a  slight  underestimate  of  the  true  reliability. 

Three  average  scores — Acquaintance,  'Suitability,  and  the  Composite — 
were  computed  for  each  ratee  for  each  rater  group.  Descriptive  statistics 
and  intercorrelations  were  developed  for  the  six  rating  scores  and  board 
rank. 

Four  one-way  analyses  of  variance  were  computed,  with  selection 
board  action  being  the  independent  variable  and  rating  scores  the 
dependent  variables. 


^-'Ebel,  R.  L.  Estimation  of  the  reliability  of  ratings.  Psychometrika, 
1951,  16,  407-424. 


RESULTS 


RELIABILITY 

The  number  of  raters  and  the  mean  scores  for  each  officer  in  each 
sample  were  compiled.  Table  1  is  the  summary  of  the  number  of  ratings 
each  ratee  received  in  each  sample.  The  percentage  of  ratees  being 
rated  by  none  or  only  one  rater  is  43.4%  for  the  AWC  and  34.8%  for 
C&GSC.  Note  that  if  each  ratee  were  to  be  rated  by  5  raters,  a  more 
acceptable  number,  the  raters  from  AWC  would  have  had  to  rate  175  of¬ 
ficers  each  and  C&GSC  raters  43  officers  each. 


Table  1 

NUMBER  OF  RATEES  BROKEN  DOWN  BY  NUMBER  OF  RATINGS  RECEIVED  IN 

AWC  AND  C&GSC 


Number  of  Raters 

Number 

AWC 

of  Ratees 

C&GSC 

0 

39i 

312 

1 

467 

377 

2 

345 

307 

3 

245 

241 

4 

165 

185 

5 

107 

132 

+5 

258 

424 

TOTAL 

1978 

1978 

The  reliability  of  averages  was  computed  for  the  suitability  and 
acquaintanceship  scales  from  each  rater  (school)  sample  using  ratees 
receiving  two  or  more  ratings.  The  reliability  index  (r^j)  for  suita¬ 
bility  scores  was  .38  and  .24  for  the  C&GSC  and  AWC  samples  respectively. 
Correcting  the  AWC  index  of  r^  ■  .24  for  the  fewer  number  of  raters  re¬ 
sults  in  a  comparable  figure  with  the  C&GSC  sample.  The  acquaintanceship 
reliability  (r^)  was  .12  and  .17  respectively  for  the  C&GSC  and  AWC 
samples.  These  indices  are  low  and  would  seem  to  reflect  the  small  num¬ 
ber  of  raters  per  ratee. 
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SCALE  SCORE  RELATIONSHIP 

For  any  further  analysis  only  officers  with  two  or  more  ratings 
from  both  rating  samples  were  included.  Table  2  gives  the  percentages 
of  officers  included  and  excluded  from  the  analysis,  broken  down  by 
board  action.  The  chi  square  of  28.86  was  significant  at  less  than  a 
probability  of  .001  and  indicates  that  a  disproportionate  share  of 
officers  considered  by  the  board  had  received  two  or  more  ratings  and 
were  therefore  included  in  the  study.  This  would  indicate  a  more 
widespread  knowledge  of  these  officers  by  the  raters. 


Table  2 

PERCENTAGE  OF  RATEES  INCLUDED  AND  EXCLUDED  FROM 
ANALYSIS,  BROKEN  DOWN  BY  BOARD  ACTION 

[Note:  383  individuals  received  less  than  two  ratings  from  both  schools] 


Board  Action 

g 

Group  1 

Group  2^ 

Total 

Not  Considered 

lO'-l  (61%) 

668  (39%) 

1719  (100%) 

Not  Selected 

75  (46%) 

89  (54%) 

164  (100%) 

Selected 

38  (40%) 

57  (60%) 

95  (100%) 

Total 

1164 

814 

1978  X2=28.86* 

*  p<  .001 

^Group  1  =  ratees  with  less  than  two  ratings  from  either  school  -  Excluded 
Group  2  =  ratees  with  two  or  more  ratings  from  both  schools  -  Included 


Table  3  gives  the  means,  standard  deviations  and  intercorrelations  of 
the  scores  from  each  sample.  Only  officers  receiving  two  or  more  ratings 
from  both  samples  (N  -  814)  were  used.  A  smali  but  significant  positive 
relationship  was  found  between  degree  of  acquaintanceship  and  suitability 
scores  (r=.19**  and  r**.07*)..  Acquaintanceship  had^a  significant  relation¬ 
ship  with  the  composite  scores  (r-.12**  and  r*.13  )  as  compared  to  the 

extreiggly  high  correlation  of  suitability  and  the  composite  (r=.97**  and 
r*.98  ).  The  formula  for  the  composite  score  dictates  the  latter  re¬ 

lationship,  and  the  former  relationship  is  a  function  of  the  correlation 
between  suitability  and  acquaintanceship. 

**  p  <  .01 

*  p<  .05 
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Table  3 


.DESCRIPTIVE  STATISTICS  AND  INTERCORRELATIONS  OF  THE  SIX 
SCALE  SCORES  FROM  THE  SAME  OFFICERS  RATED  AT  BOTH  AWC  AND  C&GSC 

(N=814) 


Variable® 

Mean 

S.D. 

Inter-r' s 

A1 

3.26 

.67 

*1 

S1 

2.70 

.92 

.19** 

S1 

C1 

5.48 

2.15 

.12** 

.97** 

C1 

A2 

3.54 

.64 

.05 

H 

O 

• 

t 

-.00  A2 

S2 

3.42 

.89 

.05 

.29** 

.29**  .07*  S2 

C2 

7.06 

2.21 

.06 

.28** 

.29**  .13**  .98**  C2 

A  »  Acquaintanceship 
S  ■  Suitability 
C  ■  Composite 

1  *  Army  War  College  Sample 

2  ■>  Command  and  General  Staff  College  Sample 

*  p  <  .  05 

**p  < . 01 


The  acquaintanceship  scores  were  not  related  to  each  other  or  to 
other  measures  across  samples  (r**.05).  Suitability  scores  were  moderate¬ 
ly  related  to  each  other  and  to  the  composite  scores  across  samples  (r=.29**) 

Table  4  details  the  means,  standard  deviations,  and  intercorrelations 
of  scores  (both  samples)  and  selection  board  rank  received;  only  officers 
selected  for  senior  service  school  attendance  were  used  (N“57) .  The  pat¬ 
tern  of  relationship  between  scale  scores  was  the  same  for  this  group  as 
for  the  total  group.  Two  correlations  with  board  rank  were  significant, 
those  with  the  suitability  and  composite  scores  from  the  AWC  raters.  Note 


**  p<  .01 
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that  the  negative  correlations  indicated  a  positive  relationship  because 
a  low  ranking  indicated  the  best  officer.  Ratings  for  C&GSC  were  not 
significantly  related  to  board  rank.  There  was  little,  if  any,  shrink¬ 
age  in  variance  for  the  selected  groups,  and  therefore  correction  for 
restriction  of  range  was  not  deemed  necessary.  These  results  indicate 
that,  first,  the  amount  of  interaction  between  rater-ratee  was  not  a 
major  contributor  to  the  suitability  scores.  Second,  the  ratings  from 
AWC  but  not  C&GSC  were  related  to  board  action.  This  indicated  some 
common  basis  for  evaluating  officer  performance  for  AWC  raters  (peers) 
and  Selection  Board  Officers.  Given  the  small  number  of  officers  in 
the  selected  group,  this  is  an  encouraging  finding. 


Table  4 


DESCRIPTIVE  STATISTICS  AND  INTERCORRELATIONS  OF  SIX  SCALE 
SCORES  AND  BOARD  RANK  OF  INDIVIDUALS  SELECTED  FOR  SENIOR  SERVICE  COLLEGE 

(N=57) 


Variable® 

Mean 

S.D. 

Inter- 

-r '  s 

A1 

3.14 

.72 

A1 

S1 

3.66 

.84 

.33* 

S1 

C1 

7.81 

2.03' 

.33* 

.98** 

C1 

A2 

3.43 

.58 

-.08 

.10 

.10 

A2 

S2 

3.95 

.74 

-.07 

.33* 

.35* 

.31* 

S2 

C2 

8.23 

1.85 

-.06 

.34** 

.29* 

.48** 

.96**  C2 

Rank 

47.25 

29.89 

-.12 

-.29* 

-.29* 

.16 

- . 04  .01  Rank 

A  *  Acquaintanceship 
B  ■  Suitability 
C  "  Composite 

1  ■  Army  War  College  Sample 

2  •  Command  and  General  Staff  College  Sample 
Rank  ■  Selection  Board  Ranking 

*  p  <  .05 

**  p  <  .01 
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MEAN  DIFFERENCES  FOR  SELECT  SUB-SAMPLES 


•Table  5  summarizes  the  means  and  standard  deviations  for  the  three 
scales  broken  down  by  school  sample  and  board  action.  Three  board  action 
sub-samples  were  defined.  The  "not  considered"  group  were  officers  elig¬ 
ible  to  attend  but  not  considered  by  the  board.  (An  unknown  pre-selection 
procedure  was  used.)  The  "not  selected"  group  were  individuals  consid¬ 
ered  by  the  board  but  not  selected.  Finally,  the  selected  group  were 
officers  selected  for  attendance  at  a  Senior  Service  School.  Again,  only 
officers  with  two  or  more  ratings  from  both  samples  were  used. 


Table  6  gives  the  analysis  of  variances  for  each  score  using  the 
above  breakdowns.  There  were  no  significant  differences  found  for  the 
acquaintanceship  scores.  Suitability  scores  for  the  favorable  action 
board  groups  (both  AWC  and  C&GSC)  were  progressively  higher,  "selected 
group"  greater  than  "considered"  and  "not  selected"  greater  than  "not 
considered."  These  results  for  both  schools  reinforce  the  previous 
positive  finding  between  rank  and  ratings  for  AWC,  by  again  indicating 
a  positive  relationship  between  the  evaluation  of  officers  by  associates 
and  the  selection  process.  The  cj2s-4/  reported  in  Table  6  represent  the 
total  variance  accounted  for  in  the  dependent  variable  by  the  independent 
variable.  The  Qf  ,121  for  AWC  is  comparable  to  a  correlation  of  .35 
and  the  of  .043  for  C&GSC  to  a  correlation  of  .21.  The  composite 
scores,  being  a  linear  combination  of  suitability  scores,  would  produce 
the  same  findings.  Higher  suitability  scores  produced  by  the  C&GSC  raters 
would  seem  to  indicate  a  more  lenient  basis  for  evaluating  performance  for 
this  group. 


SUMMARY 

Associate  ratings  from  two  samples  of  raters  were  obtained  on 
officers  eligible  for  Senior  Service  School  attendance.  Acquaintance¬ 
ship  and  suitability  ratings  were  collected  and  combined  into  a  composite 
score.  The  reliabilities  for  both  the  suitability  and  acquaintanceship 
scores  were  found  to  be  very  low  (.38  and  .24  for  suitability  and  .12  and 
.17  for  acquaintanceship).  If  these  reliabilities  were  increased,  then 
the  relationships  found  between  suitability  ratings  and  board  action 
would  become  stronger.  For  raters  receiving  two  or  more  ratings  from 
each  school  it  was  found  that:  1)  acquaintanceship  and  suitability 
scores  were  moderately  related  to  each  other  within  a  sample  and  not  re¬ 
lated  across  schools  (samples);  2)  the  suitability  scores  were  moder¬ 
ately  related  between  schools  (r**.29)  and  that  suitability  scores  for 
the  AWC  sample  were  related  to  board  ranks  assigned  to  selected  officers 


Hay 8,  W.  L.  Statistics.  Holt,  Rinehard  and  Winston:  New  York,  1963. 
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Table 


Table  6 


ANALYSIS  OF  VARIANCE  FOR  EACH  SCHOOL  AND  ACROSS  BOARD  ACTION 
FOR  ACQUAINTANCESHIP  AND  SUITABILITY  SCORES 


AWC 

Acquaintanceship 


Source 

SS 

df 

MS 

F 

Board  Action 

2.44 

2 

1.22 

2.71 

Error 

366.89 

811 

.45 

Suitability 

Source 

SS 

df 

MS 

F 

Board  Action 

85.60 

2 

42.80 

57.07** 

Error 

604.57 

811 

.75 

,..2  _  in. 

* 

C&GSC 

Acquaintanceship 

Source 

SS 

df 

MS 

F 

Board  Action 

1.12 

2 

.56 

1.37 

Error 

330.32 

811 

.41 

Suitability 

Source 

SS 

df 

MS 

F 

Board  Action 

29.52 

2 

14.76 

19.47** 

Error 

617.86 

811 

.76 

oi2  =  .043 

**p  <  . 01 
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(r=-.29);  3)  the  C&GSC  sample  produced  higher  suitability  scores  than 
the  AWC  sample;  and  4)  the  officers  selected  by  the  board  were  rated 
more  -  suitable  as  a  group  by  associates  than  officers  not  selected  and  not 
considered.  There  was  agreement  between  the  board  and  raters  on  the  most 
qualified  officer.  These  results  indicate  that  the  inclusion  of  an  asso¬ 
ciate  rating  score  as  part  of  the  selection  process  would  add  new  sources 
of  information  which  are  not  totally  at  variance  with  present  procedures. 
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APPENDIXES 


APPENDIX  A  -  SURVEY  INSTRUCTIONS 


PLEASE  ENTER  YOUR 

STUDENT  BOX  NO. _ 

AND  GRADE:  0- 


MESSAGE  CENTER:  RETURN  TO 

COL  D.  H.  WARDROP 
Box  191,  USAWC 


SURVEY:  SELECTION  FOR  SENIOR  SERVICE  COLLEGE 


The  objective  of  this  survey  is  to  get  an  indication  of  the  possible  use 
of  peer  ratings  for  assisting  in  the  selection  of  Army  officers  for 
attendance  at  a  senior  service  college.  For  the  purpose  of  this  survey 
you  are  considered  a  peer  of  every  officer  on  the  enclosed  roster. 

The  enclosed  roster  contains  all  infantry  officers  in  the  zone  of  eligi¬ 
bility  to  be  selected  to  attend  one  of  the  senior  service  colleges  during 
the  academic  year  1971-1972.  Request  you  perform  the  following  three 
actions  regarding  this  roster: 

1.  As  step  #1:  Underline  each  name  that  you  are  reasonably  sure  you 
recognize.  Then  perform  the  next  two  actions  jointly  as  step  i/2. 

2.  ASSOCIATION:  There  are  innumerable  ways  one  person  can  "know" 
another.  The  scale  below  does  not  include  all  possible  variations,  but 
rather  its  purpose  is  to  provide  an  indication  of  how  well  you  know  the 
individual  concerned.  From  the  scale,  please  select  the  choice  which 
best  represents  your  association  with  the  individual  whose  names  you  have 
underlined Do  not  be  overly  concerned  with  how  long  the  relationship 
has  existed.  (For  each  name  underlined,  enter  appropriate  number  (1-5) 
in  "Association"  column.) 

I  know  individual  by: 

1.  Reputation  and/or  1-2  personal  contacts. 

2.  Minimum  social  and/or  professional  contact. 

3.  Occasional  social  and/or  professional  contact. 

4.  Frequent  social  and/or  professional  contact. 


5.  Close  and  frequent  social  and/or  professional  contact. 

'3.  SUITABILITY:  In  this  operation,  please  indicate  your  opinion  of 
the  individual’s  suitability  for  attendance  at  a  senior  service  college. 


APPENDIX  A  (continued) 


Do  not  consider  how  well  you  know  the  person — this  will  be  accounted  for 
in  the  association  scale  above.  As  a  general  criterion,  consider  that 
those  who  attend  the  senior  service  colleges  will  probably  move  upward 
in  the  Army  to  positions  of  greater  responsibility.  (For  each  name  under¬ 
lined,  enter  appropriate  number  (1-5)  in  "Suitability"  column.) 

In  my  opinion,  this  individual  should  be: 

1.  Not  selected. 

2.  Selected  later  if  warranted  by  records. 

3.  Selected  next  year. 

A.  Selected  as  an  alternate  or  standby  this  year. 

5.  Selected  immediately, 


Please  use  the  following  guidance  in  completing  your  roster: 

1.  There  is  no  limit  on  the  number  of  officers  you  select  for  any 
year,  or  reject. 

2.  There  are  no  "tricks"  or  hidden  meanings  in  this  survey.  Complete 
the  roster  in  a  judicious,  straightforward  manner. 

3.  If  you  are  aware  of  any  personal  desires  of  the  officer  under  con¬ 
sideration,  do  not  take. these  into  account.  Make  a  selection  or  rejection 
based  on  the  officer's  professional  qualifications  and  potential  as  you 
know  them. 

I  have  requested  your  box  number  on  the  first  page  of  the  survey  for  Ay 
use  in  case  of  questions  after  you  return  it  to  me.  I  shall  treat  your 
completed  response  as  confidential  and  request  you  also  treat  it  in  a 
confidential  manner.  Please  do  not  discuss  your  response  with  other 
students.  As  information  from  your  response  is  transferred  to  data  cards, 
your  identification  with  the  roster  will  cease,  and  the  names  of  the 
individuals  on  the  roster  will  be  dropped  and  will  be  represented  by  a 
code  number  in  all  scoring,  analysis,  and  presentations  of  results. 

Thank  you. 
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APPENDIX  B 


COMPOSITE  SCORES  FOR  ALL  COMBINATIONS  OF  ACQUAINTANCESHIP 
AND  SUITABILITY  SCORES 


n 

Formula:  Composite  =  (s-3) ( .5) (a+l)+6 

1  _ 


SUITABILITY  ACQUAINTANCESHIP 

SCORE  SCORE 


COMPOSITE 

SCORE 


